Free-Viewpoint Video from Depth Cameras

Depth cameras, which provide color and depth information per pixel at video rates, offer exciting new opportunities in computer graphics. We address the challenge of supporting free-viewpoint video of dynamic 3D scenes using live data captured and streamed from widely-spaced viewpoints by a handful of synchronized depth cameras. We introduce the concept of the depth hull, which is a generalization of the well-known visual hull. The depth hull reflects all the dense depth information as observed from several centers of projection around the scene. It is the best approximation of the scene geometry that can be obtained from a given set of depth camera recordings. We first present a general improvement to the best existing visual hull rendering algorithm, which is of independent interest. We then use this to contribute a hardware-accelerated method for rendering novel views from depth hulls in real-time. This method is based on a combination of techniques from projective shadow mapping and constructive solid geometry (CSG). Our rendering method achieves high-quality results even when only a modest number of depth cameras are deployed. They are applicable to any set of images with accompanying dense depth maps that correspond to arbitrary viewing positions around the scene. We provide experimental results using a system incorporating two depth cameras recording a dynamic scene. To the best of our knowledge, these are the first results on free-viewpoint image synthesis using commercially-available depth cameras.